{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5d27b550",
   "metadata": {},
   "source": [
    "# Evaluating with Bootstrap Metrics\n",
    "\n",
    "Bootstrap metrics can help us to more thoroughly evaluate a trading strategy, as we will see in this notebook.\n",
    "\n",
    "[In the previous notebook](https://www.pybroker.com/en/latest/notebooks/2.%20Backtesting%20a%20Strategy.html), we implemented a trading strategy and backtested it. Here is the implementation again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8b5eb791",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pybroker\n",
    "from pybroker import Strategy, StrategyConfig, YFinance\n",
    "\n",
    "pybroker.enable_data_source_cache('my_strategy')\n",
    "\n",
    "def buy_low(ctx):\n",
    "    if ctx.long_pos():\n",
    "        return\n",
    "    if ctx.bars >= 2 and ctx.close[-1] < ctx.low[-2]:\n",
    "        ctx.buy_shares = ctx.calc_target_shares(0.25)\n",
    "        ctx.buy_limit_price = ctx.close[-1] - 0.01\n",
    "        ctx.hold_bars = 3\n",
    "        \n",
    "def short_high(ctx):\n",
    "    if ctx.short_pos():\n",
    "        return\n",
    "    if ctx.bars >= 2 and ctx.close[-1] > ctx.high[-2]:\n",
    "        ctx.sell_shares = 100\n",
    "        ctx.hold_bars = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8eec3c9",
   "metadata": {},
   "source": [
    "As before, we create a new [Strategy](https://www.pybroker.com/en/latest/reference/pybroker.strategy.html#pybroker.strategy.Strategy) instance with the given configurations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4ceca2ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "config = StrategyConfig(initial_cash=500_000, bootstrap_sample_size=100)\n",
    "strategy = Strategy(YFinance(), '3/1/2017', '3/1/2022', config)\n",
    "strategy.add_execution(buy_low, ['AAPL', 'MSFT'])\n",
    "strategy.add_execution(short_high, ['TSLA'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94664fc8",
   "metadata": {},
   "source": [
    "This time, the ```Strategy``` is configured with a [bootstrap_sample_size](https://www.pybroker.com/en/latest/reference/pybroker.config.html#pybroker.config.StrategyConfig.bootstrap_sample_size) of ```100``` because of the small amount of historical data being used. Next, we run the backtest with bootstrap metrics enabled:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4316630b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Backtesting: 2017-03-01 00:00:00 to 2022-03-01 00:00:00\n",
      "\n",
      "Loaded cached bar data.\n",
      "\n",
      "Test split: 2017-03-01 00:00:00 to 2022-02-28 00:00:00\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1259 of 1259) |####################| Elapsed Time: 0:00:00 Time:  0:00:00\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Calculating bootstrap metrics: sample_size=100, samples=10000...\n",
      "Calculated bootstrap metrics: 0:00:03 \n",
      "\n",
      "Finished backtest: 0:00:05\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>trade_count</td>\n",
       "      <td>388.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>initial_market_value</td>\n",
       "      <td>500000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>end_market_value</td>\n",
       "      <td>655753.670000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>total_pnl</td>\n",
       "      <td>156575.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>unrealized_pnl</td>\n",
       "      <td>-821.330000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>total_return_pct</td>\n",
       "      <td>31.315000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>total_profit</td>\n",
       "      <td>383032.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>total_loss</td>\n",
       "      <td>-226457.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>total_fees</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>max_drawdown</td>\n",
       "      <td>-30181.580000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>max_drawdown_pct</td>\n",
       "      <td>-4.554816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>win_rate</td>\n",
       "      <td>52.577320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>loss_rate</td>\n",
       "      <td>47.422680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>winning_trades</td>\n",
       "      <td>204.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>losing_trades</td>\n",
       "      <td>184.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>avg_pnl</td>\n",
       "      <td>403.543814</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>avg_return_pct</td>\n",
       "      <td>0.279639</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>avg_trade_bars</td>\n",
       "      <td>2.414948</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>avg_profit</td>\n",
       "      <td>1877.609804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>avg_profit_pct</td>\n",
       "      <td>3.168775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>avg_winning_trade_bars</td>\n",
       "      <td>2.465686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>avg_loss</td>\n",
       "      <td>-1230.746739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>avg_loss_pct</td>\n",
       "      <td>-2.923533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>avg_losing_trade_bars</td>\n",
       "      <td>2.358696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>largest_win</td>\n",
       "      <td>20797.970000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>largest_win_pct</td>\n",
       "      <td>14.490000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>largest_win_bars</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>largest_loss</td>\n",
       "      <td>-10831.630000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>largest_loss_pct</td>\n",
       "      <td>-6.490000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>largest_loss_bars</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>max_wins</td>\n",
       "      <td>7.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>max_losses</td>\n",
       "      <td>7.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>sharpe</td>\n",
       "      <td>0.054488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>sortino</td>\n",
       "      <td>0.061320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>profit_factor</td>\n",
       "      <td>1.312935</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>ulcer_index</td>\n",
       "      <td>0.627821</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>upi</td>\n",
       "      <td>0.035531</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>equity_r2</td>\n",
       "      <td>0.893202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>std_error</td>\n",
       "      <td>63596.828230</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      name          value\n",
       "0              trade_count     388.000000\n",
       "1     initial_market_value  500000.000000\n",
       "2         end_market_value  655753.670000\n",
       "3                total_pnl  156575.000000\n",
       "4           unrealized_pnl    -821.330000\n",
       "5         total_return_pct      31.315000\n",
       "6             total_profit  383032.400000\n",
       "7               total_loss -226457.400000\n",
       "8               total_fees       0.000000\n",
       "9             max_drawdown  -30181.580000\n",
       "10        max_drawdown_pct      -4.554816\n",
       "11                win_rate      52.577320\n",
       "12               loss_rate      47.422680\n",
       "13          winning_trades     204.000000\n",
       "14           losing_trades     184.000000\n",
       "15                 avg_pnl     403.543814\n",
       "16          avg_return_pct       0.279639\n",
       "17          avg_trade_bars       2.414948\n",
       "18              avg_profit    1877.609804\n",
       "19          avg_profit_pct       3.168775\n",
       "20  avg_winning_trade_bars       2.465686\n",
       "21                avg_loss   -1230.746739\n",
       "22            avg_loss_pct      -2.923533\n",
       "23   avg_losing_trade_bars       2.358696\n",
       "24             largest_win   20797.970000\n",
       "25         largest_win_pct      14.490000\n",
       "26        largest_win_bars       3.000000\n",
       "27            largest_loss  -10831.630000\n",
       "28        largest_loss_pct      -6.490000\n",
       "29       largest_loss_bars       3.000000\n",
       "30                max_wins       7.000000\n",
       "31              max_losses       7.000000\n",
       "32                  sharpe       0.054488\n",
       "33                 sortino       0.061320\n",
       "34           profit_factor       1.312935\n",
       "35             ulcer_index       0.627821\n",
       "36                     upi       0.035531\n",
       "37               equity_r2       0.893202\n",
       "38               std_error   63596.828230"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = strategy.backtest(calc_bootstrap=True)\n",
    "result.metrics_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1211ffde",
   "metadata": {},
   "source": [
    "When we look at the ```total_pnl``` metric above, it seems that we have a profitable trading strategy on our first try. However, we cannot be completely sure that these results are repeatable and not just due to chance. To gain more confidence in our results, we can use the [boostrap method](https://en.wikipedia.org/wiki/Bootstrapping_(statistics)) to compute metrics.\n",
    "\n",
    "The bootstrap method works by repeatedly computing a metric on random samples drawn from the backtest's returns. Then, the metric is computed on each random sample, and the average is taken. By doing this on thousands of random samples, we obtain a more robust and accurate estimate of the metric.\n",
    "\n",
    "\n",
    "## Confidence Intervals\n",
    "\n",
    "**PyBroker** applies the bootstrap method to calculate [confidence intervals](https://en.wikipedia.org/wiki/Confidence_interval) for two performance metrics, the [Profit Factor](https://www.pybroker.com/en/latest/reference/pybroker.eval.html#pybroker.eval.EvalMetrics.profit_factor) and [Sharpe Ratio](https://en.wikipedia.org/wiki/Sharpe_ratio):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4ff0d074",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>lower</th>\n",
       "      <th>upper</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>name</th>\n",
       "      <th>conf</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">Profit Factor</th>\n",
       "      <th>97.5%</th>\n",
       "      <td>0.594243</td>\n",
       "      <td>4.400753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95%</th>\n",
       "      <td>0.719539</td>\n",
       "      <td>3.715684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90%</th>\n",
       "      <td>0.877060</td>\n",
       "      <td>3.153457</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">Sharpe Ratio</th>\n",
       "      <th>97.5%</th>\n",
       "      <td>-0.136541</td>\n",
       "      <td>0.243573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95%</th>\n",
       "      <td>-0.100146</td>\n",
       "      <td>0.220099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90%</th>\n",
       "      <td>-0.060583</td>\n",
       "      <td>0.193326</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        lower     upper\n",
       "name          conf                     \n",
       "Profit Factor 97.5%  0.594243  4.400753\n",
       "              95%    0.719539  3.715684\n",
       "              90%    0.877060  3.153457\n",
       "Sharpe Ratio  97.5% -0.136541  0.243573\n",
       "              95%   -0.100146  0.220099\n",
       "              90%   -0.060583  0.193326"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.bootstrap.conf_intervals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0c638c7",
   "metadata": {},
   "source": [
    "**PyBroker** uses the [bias corrected and accelerated (BCa) bootstrap method](https://blogs.sas.com/content/iml/2017/07/12/bootstrap-bca-interval.html) to calculate the confidence intervals for these metrics. The returns are sampled per-bar rather than per-trade to capture more information in the metrics.\n",
    "\n",
    "The resulting table shows the lower bound of the confidence interval at the given confidence level. This provides a more conservative estimate of the strategy's performance. For example, we can be ```97.5%``` confident that the Sharpe Ratio is at or above a given value of *x*. \n",
    "\n",
    "In this example, the Sharpe Ratio has negative lower bounds, and the lower bounds of the Profit Factor are less than 1, which suggests that the strategy is not reliable.\n",
    "\n",
    "## Maximum Drawdown\n",
    "\n",
    "In this section, we examine the maximum drawdown of the strategy using the bootstrap method. The probabilities of the drawdown not exceeding certain values, represented in cash and percentage of portfolio equity, are displayed below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9953d460",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>amount</th>\n",
       "      <th>percent</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>conf</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>99.9%</th>\n",
       "      <td>-66401.25</td>\n",
       "      <td>-10.462527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99%</th>\n",
       "      <td>-50062.49</td>\n",
       "      <td>-7.963632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95%</th>\n",
       "      <td>-35794.82</td>\n",
       "      <td>-5.848482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90%</th>\n",
       "      <td>-29931.10</td>\n",
       "      <td>-4.912087</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         amount    percent\n",
       "conf                      \n",
       "99.9% -66401.25 -10.462527\n",
       "99%   -50062.49  -7.963632\n",
       "95%   -35794.82  -5.848482\n",
       "90%   -29931.10  -4.912087"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.bootstrap.drawdown_conf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "408285d5",
   "metadata": {},
   "source": [
    "These confidence levels were obtained using per-bar returns from the backtest's out-of-sample results, similar to how the Profit Factor and Sharpe Ratio were calculated.\n",
    "\n",
    "We can observe that the bootstrapped max drawdown of ```-10.46%``` at a ```99.9%``` confidence level is worse than the ```-4.55%``` we saw in our original results. This highlights the importance of using randomized tests to evaluate the performance of your trading strategy.\n",
    "\n",
    "[In the next notebook, we will discuss how to incorporate ranking and position sizing in your trading strategies](https://www.pybroker.com/en/latest/notebooks/4.%20Ranking%20and%20Position%20Sizing.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
